Getting started with xdatasets

The xdatasets library enables users to effortlessly access a vast collection of earth observation datasets that are compatible with xarray formats.

The library adopts an opinionated approach to data querying and caters to the specific needs of certain user groups, such as hydrologists, climate scientists, and engineers. One of the functionalities of xdatasets is the ability to extract data at a specific location or within a designated region, such as a watershed or municipality, while also enabling spatial and temporal operations.

To use xdatasets, users must employ a query. A straightforward query to extract the variables t2m (2m temperature) and tp (Total precipitation) from the era5_reanalysis_single_levels dataset at two geographical positions (Montreal and Toronto) could be as follows:

query = {
    "variables": {"era5_reanalysis_single_levels": ["t2m", "tp"]},
    "space": {
        "clip": "point", # bbox, point or polygon
        "geometry": {'Montreal' : (45.508888, -73.561668),
                     'Toronto' : (43.651070, -79.347015)
                    }
    }
}

An example of a more complex query would look like this. This query calls the same variables as above. However, instead of specifying geographical positions, GeoPandas.DataFrame is used to provide features (such as shapefiles or geojson) for extracting data within each of them. Each polygon is identified using the unique identifier Station, and a spatial average is computed within each one (aggregation: True). The dataset, initially at an hourly time step, is converted into a daily time step while applying one or more temporal aggregations for each variable as prescribed in the query. The xdatasets function ultimately returns the dataset for the specified date range and time zone.

query = {
    "variables": {"era5_reanalysis_single_levels": ["t2m", "tp"]},
    "space": {
        "clip": "polygon", # bbox, point or polygon
        "aggregation": False,
        "geometry": gdf,
        "unique_id": "Station"
    },
    "time": {
        "timestep": "D",
        "aggregation": {"tp": np.nansum,
                        "t2m": [np.nanmax, np.nanmin]},

        "start": '1959-01-01',
        "end": '1961-05-31',
        "timezone": 'America/Montreal',
    },
}

Don't worry! Additional examples below will help in comprehending the range of possible queries.

Note

Don’t worry! Additional examples below will help in comprehending the range of possible queries.

## Query climate datasets

import xdatasets as xd
import intake
import numpy as np
import geopandas as gpd
import pandas as pd
from pathlib import Path

bucket = Path('https://s3.us-east-2.wasabisys.com/watersheds-polygons/MELCC/json')

paths = [bucket.joinpath('023003/023003.json'),
         bucket.joinpath('031101/031101.json'),
         bucket.joinpath('040111/040111.json')]

gdf = pd.concat([gpd.read_file(path).to_crs(4326) for path in paths]).reset_index(drop=True)
gdf

	Station	Superficie	geometry
0	023003	208.4591919813271	POLYGON ((-70.82601 46.81658, -70.82728 46.815...
1	031101	111.7131058782722	POLYGON ((-73.98519 45.21072, -73.98795 45.209...
2	040111	433.440893903503	POLYGON ((-74.06645 46.02253, -74.06647 46.022...

gdf.hvplot(geo=True,
           tiles='ESRI',
           color='Station',
           alpha=0.8,
           width=750,
           height=450,
           legend='top',
           hover_cols=['Station','Superficie'])

%%time

# http://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases

query = {
    "variables": {"era5_reanalysis_single_levels": ["t2m", "tp"]},
    "space": {
        "clip": "polygon", # bbox, point or polygon
        "aggregation": False,
        "geometry": gdf,
        "unique_id": "Station"
    },
    "time": {
#         "timestep": "D",
#         "aggregation": {"tp": np.nansum,
#                         "t2m": [np.nanmax, np.nanmin]},

        "start": '1979-01-01',
        "end": '2020-12-31',
#         "timezone": 'America/Montreal',
    },
}


xds = xd.Dataset(**query)

3it [00:26,  8.90s/it]

<xarray.Dataset>
Dimensions:    (latitude: 12, longitude: 13, time: 368184, Station: 3)
Coordinates:
  * latitude   (latitude) float32 44.5 44.75 45.0 45.25 ... 46.75 47.0 47.25
  * longitude  (longitude) float32 -75.0 -74.75 -74.5 ... -70.75 -70.5 -70.25
  * time       (time) datetime64[ns] 1979-01-01 ... 2020-12-31T23:00:00
Dimensions without coordinates: Station
Data variables:
    t2m        (Station, time, latitude, longitude) float32 nan nan ... nan nan
    tp         (Station, time, latitude, longitude) float32 nan nan ... nan nan
Attributes:
    institution:  ECMWF
    source:       Reanalysis
    title:        ERA5 forecasts
CPU times: user 13.7 s, sys: 5.41 s, total: 19.1 s
Wall time: 28.1 s

ds_clipped = xds.bbox_clip(xds.data.isel(Station=0))
ds_clipped

<xarray.Dataset>
Dimensions:    (time: 368184, latitude: 6, longitude: 6)
Coordinates:
  * latitude   (latitude) float32 46.0 46.25 46.5 46.75 47.0 47.25
  * longitude  (longitude) float32 -71.5 -71.25 -71.0 -70.75 -70.5 -70.25
  * time       (time) datetime64[ns] 1979-01-01 ... 2020-12-31T23:00:00
Data variables:
    t2m        (time, latitude, longitude) float32 272.8 272.1 ... 269.3 269.1
    tp         (time, latitude, longitude) float32 nan nan nan ... 0.0 0.0 0.0
Attributes:
    institution:  ECMWF
    source:       Reanalysis
    title:        ERA5 forecasts

xarray.Dataset

• Dimensions:
  • time: 368184
  • latitude: 6
  • longitude: 6
• Coordinates: (3)
  • latitude
    (latitude)
    float32
    46.0 46.25 46.5 46.75 47.0 47.25

    long_name :

    latitude

    standard_name :

    latitude

    units :

    degrees_north

    array([46.  , 46.25, 46.5 , 46.75, 47.  , 47.25], dtype=float32)

  • longitude
    (longitude)
    float32
    -71.5 -71.25 -71.0 ... -70.5 -70.25

    array([-71.5 , -71.25, -71.  , -70.75, -70.5 , -70.25], dtype=float32)

  • time
    (time)
    datetime64[ns]
    1979-01-01 ... 2020-12-31T23:00:00

    standard_name :

    time

    array(['1979-01-01T00:00:00.000000000', '1979-01-01T01:00:00.000000000',
           '1979-01-01T02:00:00.000000000', ..., '2020-12-31T21:00:00.000000000',
           '2020-12-31T22:00:00.000000000', '2020-12-31T23:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (2)
  • t2m
    (time, latitude, longitude)
    float32
    272.8 272.1 272.2 ... 269.3 269.1

    long_name :

    2 metre temperature

    nameCDM :

    2_metre_temperature_surface

    nameECMWF :

    2 metre temperature

    product_type :

    analysis

    shortNameECMWF :

    2t

    standard_name :

    air_temperature

    units :

    K

    array([[[272.75  , 272.125 , 272.25  , 271.9375, 271.4375, 270.625 ],
            [272.5   , 271.9375, 271.5625, 271.125 , 270.75  , 270.375 ],
            [270.75  , 271.0625, 270.8125, 270.3125, 269.625 , 269.6875],
            [268.8125, 268.9375, 268.8125, 268.5625, 269.125 , 269.375 ],
            [268.    , 267.4375, 267.3125, 267.3125, 268.25  , 268.8125],
            [267.625 , 267.    , 266.875 , 267.5625, 270.1875, 270.6875]],
    
           [[272.8125, 272.25  , 272.375 , 272.0625, 271.625 , 270.8125],
            [272.625 , 272.125 , 271.6875, 271.3125, 270.9375, 270.5   ],
            [270.9375, 271.25  , 270.9375, 270.5   , 269.75  , 269.75  ],
            [268.9375, 269.0625, 269.0625, 268.9375, 269.25  , 269.4375],
            [268.125 , 267.625 , 267.375 , 267.375 , 268.3125, 268.875 ],
            [267.75  , 267.1875, 267.    , 267.625 , 270.1875, 270.6875]],
    
           [[272.9375, 272.3125, 272.4375, 272.1875, 271.6875, 270.9375],
            [272.875 , 272.3125, 271.9375, 271.5625, 271.125 , 270.75  ],
            [271.25  , 271.5625, 271.25  , 270.75  , 270.    , 270.    ],
            [269.125 , 269.3125, 269.3125, 269.25  , 269.5625, 269.6875],
            [268.25  , 267.75  , 267.5   , 267.5625, 268.4375, 269.0625],
            [267.875 , 267.25  , 267.0625, 267.6875, 270.25  , 270.875 ]],
    ...
            [270.1875, 269.875 , 270.0625, 270.125 , 269.875 , 269.625 ],
            [270.875 , 270.5   , 269.75  , 269.0625, 268.1875, 268.5   ],
            [270.375 , 270.75  , 270.4375, 269.875 , 268.75  , 268.5   ],
            [267.9375, 268.1875, 269.1875, 269.25  , 269.8125, 269.625 ],
            [265.1875, 264.8125, 265.9375, 268.    , 271.25  , 271.    ]],
    
           [[267.8125, 267.9375, 268.375 , 268.4375, 268.125 , 267.8125],
            [267.8125, 267.5625, 267.75  , 267.8125, 267.5625, 267.1875],
            [268.8125, 268.6875, 268.0625, 267.375 , 266.5   , 266.5625],
            [268.6875, 269.25  , 269.    , 268.5   , 267.375 , 266.9375],
            [266.4375, 266.625 , 267.625 , 267.6875, 268.375 , 268.125 ],
            [263.5625, 263.    , 264.125 , 266.25  , 269.8125, 269.625 ]],
    
           [[267.5   , 267.6875, 268.125 , 268.25  , 268.    , 267.75  ],
            [267.625 , 267.5   , 267.8125, 267.875 , 267.625 , 267.25  ],
            [268.75  , 268.9375, 268.375 , 267.75  , 266.8125, 266.8125],
            [268.8125, 269.5   , 269.3125, 268.75  , 267.625 , 267.1875],
            [266.625 , 266.8125, 267.6875, 267.75  , 268.4375, 268.25  ],
            [263.625 , 263.0625, 264.125 , 266.0625, 269.3125, 269.125 ]]],
          dtype=float32)

  • tp
    (time, latitude, longitude)
    float32
    nan nan nan nan ... 0.0 0.0 0.0 0.0

    long_name :

    Total precipitation

    nameCDM :

    Total_precipitation_1hour_Accumulation

    nameECMWF :

    Total precipitation

    product_type :

    forecast

    shortNameECMWF :

    tp

    standard_name :

    precipitation_amount

    units :

    m

    array([[[nan, nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan, nan]],
    
           [[nan, nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan, nan]],
    
           [[nan, nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan, nan]],
    ...
           [[ 0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.]],
    
           [[ 0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.]],
    
           [[ 0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.]]], dtype=float32)

• Indexes: (3)
  • latitude
    PandasIndex

    PandasIndex(Float64Index([46.0, 46.25, 46.5, 46.75, 47.0, 47.25], dtype='float64', name='latitude'))

  • longitude
    PandasIndex

    PandasIndex(Float64Index([-71.5, -71.25, -71.0, -70.75, -70.5, -70.25], dtype='float64', name='longitude'))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['1979-01-01 00:00:00', '1979-01-01 01:00:00',
                   '1979-01-01 02:00:00', '1979-01-01 03:00:00',
                   '1979-01-01 04:00:00', '1979-01-01 05:00:00',
                   '1979-01-01 06:00:00', '1979-01-01 07:00:00',
                   '1979-01-01 08:00:00', '1979-01-01 09:00:00',
                   ...
                   '2020-12-31 14:00:00', '2020-12-31 15:00:00',
                   '2020-12-31 16:00:00', '2020-12-31 17:00:00',
                   '2020-12-31 18:00:00', '2020-12-31 19:00:00',
                   '2020-12-31 20:00:00', '2020-12-31 21:00:00',
                   '2020-12-31 22:00:00', '2020-12-31 23:00:00'],
                  dtype='datetime64[ns]', name='time', length=368184, freq=None))

• Attributes: (3)

  institution :

  ECMWF

  source :

  Reanalysis

  title :

  ERA5 forecasts

ds_clipped.t2m.hvplot()

%%time

# http://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases

query = {
    "variables": {"era5_reanalysis_single_levels": ["t2m", "tp"]},
    "space": {
        "clip": "polygon", # bbox, point or polygon
        "aggregation": True,
        "geometry": gdf,
        "unique_id": "Station"
    },
    "time": {
        "timestep": "D",
        "aggregation": {"tp": [np.nansum],
                        "t2m": [np.nanmax, np.nanmin]},

        "start": '1979-01-01',
        "end": '2020-12-31',
        "timezone": 'America/Montreal',
    },
}


xds = xd.Dataset(**query)

3it [00:30, 10.32s/it]

<xarray.Dataset>
Dimensions:     (time: 368184, Station: 3)
Coordinates:
  * time        (time) datetime64[ns] 1978-12-31T19:00:00 ... 2020-12-31T18:0...
    longitude   (Station) float64 -70.94 -74.14 -74.27
    latitude    (Station) float64 46.72 45.18 46.08
    geom        (Station) int64 0 1 2
  * Station     (Station) object '023003' '031101' '040111'
    Superficie  (Station) object '208.4591919813271' ... '433.440893903503'
Data variables:
    t2m         (time, Station) float32 268.8 274.4 269.7 ... 269.3 271.3 266.5
    tp          (time, Station) float32 nan nan nan nan nan ... 0.0 0.0 0.0 0.0
Attributes:
    institution:    ECMWF
    source:         Reanalysis
    title:          ERA5 forecasts
    timezone:       America/Montreal
    regrid_method:  conservative

Processing tp: 100%|██████████| 2/2 [00:07<00:00,  3.59s/it]

CPU times: user 22.5 s, sys: 1.26 s, total: 23.7 s
Wall time: 39 s

xds.data.to_dataframe().hvplot(x='time',
                               y=['t2m_nanmax','t2m_nanmin'],
                               grid=True,
                               width=750,
                               height=450,
                               groupby='Station',
                               legend='top',
                               widget_location='bottom')

xds.data.to_dataframe().hvplot(x='time',
                               y=['tp_nansum'],
                               grid=True,
                               width=750,
                               height=450,
                               groupby='Station',
                               legend='top',
                               widget_location='bottom')

xds.data

<xarray.Dataset>
Dimensions:     (Station: 3, time: 15342)
Coordinates:
    longitude   (Station) float64 -70.94 -74.14 -74.27
    latitude    (Station) float64 46.72 45.18 46.08
    geom        (Station) int64 0 1 2
  * Station     (Station) object '023003' '031101' '040111'
    Superficie  (Station) object '208.4591919813271' ... '433.440893903503'
  * time        (time) datetime64[ns] 1978-12-31 1979-01-01 ... 2020-12-31
Data variables:
    t2m_nanmax  (time, Station) float32 271.1 276.4 271.5 ... 273.7 276.1 272.7
    t2m_nanmin  (time, Station) float32 268.8 274.4 269.7 ... 269.0 271.3 266.3
    tp_nansum   (time, Station) float32 0.0 0.0 0.0 ... 0.002386 0.002362
Attributes:
    institution:  ECMWF
    source:       Reanalysis
    title:        ERA5 forecasts
    timezone:     America/Montreal

xarray.Dataset

• Dimensions:
  • Station: 3
  • time: 15342
• Coordinates: (6)
  • longitude
    (Station)
    float64
    -70.94 -74.14 -74.27

    array([-70.94448103, -74.14353175, -74.27460773])

  • latitude
    (Station)
    float64
    46.72 45.18 46.08

    array([46.71517431, 45.17637767, 46.08306936])

  • geom
    (Station)
    int64
    0 1 2

    array([0, 1, 2])

  • Station
    (Station)
    object
    '023003' '031101' '040111'

    array(['023003', '031101', '040111'], dtype=object)

  • Superficie
    (Station)
    object
    '208.4591919813271' ... '433.440...

    array(['208.4591919813271', '111.7131058782722', '433.440893903503'],
          dtype=object)

  • time
    (time)
    datetime64[ns]
    1978-12-31 ... 2020-12-31

    array(['1978-12-31T00:00:00.000000000', '1979-01-01T00:00:00.000000000',
           '1979-01-02T00:00:00.000000000', ..., '2020-12-29T00:00:00.000000000',
           '2020-12-30T00:00:00.000000000', '2020-12-31T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (3)
  • t2m_nanmax
    (time, Station)
    float32
    271.1 276.4 271.5 ... 276.1 272.7

    long_name :

    2 metre temperature

    nameCDM :

    2_metre_temperature_surface

    nameECMWF :

    2 metre temperature

    product_type :

    analysis

    shortNameECMWF :

    2t

    standard_name :

    air_temperature

    units :

    K

    array([[271.1253 , 276.40222, 271.52487],
           [278.8807 , 281.0484 , 277.00375],
           [277.17114, 276.26514, 273.84592],
           ...,
           [272.68164, 270.683  , 264.64752],
           [269.04025, 275.22174, 270.5143 ],
           [273.73267, 276.10953, 272.7495 ]], dtype=float32)

  • t2m_nanmin
    (time, Station)
    float32
    268.8 274.4 269.7 ... 271.3 266.3

    long_name :

    2 metre temperature

    nameCDM :

    2_metre_temperature_surface

    nameECMWF :

    2 metre temperature

    product_type :

    analysis

    shortNameECMWF :

    2t

    standard_name :

    air_temperature

    units :

    K

    array([[268.78976, 274.38742, 269.73624],
           [271.37527, 274.3489 , 271.82175],
           [269.25568, 263.26852, 260.843  ],
           ...,
           [263.42   , 265.07678, 257.20084],
           [260.9095 , 264.93744, 258.1872 ],
           [268.95377, 271.25836, 266.3292 ]], dtype=float32)

  • tp_nansum
    (time, Station)
    float32
    0.0 0.0 0.0 ... 0.002386 0.002362

    long_name :

    Total precipitation

    nameCDM :

    Total_precipitation_1hour_Accumulation

    nameECMWF :

    Total precipitation

    product_type :

    forecast

    shortNameECMWF :

    tp

    standard_name :

    precipitation_amount

    units :

    m

    array([[0.00000000e+00, 0.00000000e+00, 0.00000000e+00],
           [1.63621679e-02, 1.32727744e-02, 2.14345567e-02],
           [1.62692089e-02, 5.28216781e-03, 3.99874523e-03],
           ...,
           [7.54918088e-04, 1.28116096e-06, 0.00000000e+00],
           [5.90828713e-03, 6.06924109e-03, 1.17766075e-02],
           [4.65553068e-03, 2.38613598e-03, 2.36194278e-03]], dtype=float32)

• Indexes: (2)
  • Station
    PandasIndex

    PandasIndex(Index(['023003', '031101', '040111'], dtype='object', name='Station'))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['1978-12-31', '1979-01-01', '1979-01-02', '1979-01-03',
                   '1979-01-04', '1979-01-05', '1979-01-06', '1979-01-07',
                   '1979-01-08', '1979-01-09',
                   ...
                   '2020-12-22', '2020-12-23', '2020-12-24', '2020-12-25',
                   '2020-12-26', '2020-12-27', '2020-12-28', '2020-12-29',
                   '2020-12-30', '2020-12-31'],
                  dtype='datetime64[ns]', name='time', length=15342, freq='D'))

• Attributes: (4)

  institution :

  ECMWF

  source :

  Reanalysis

  title :

  ERA5 forecasts

  timezone :

  America/Montreal